Tree-structured Neural Network Research Articles

Tree-Structured Neural Network for Hyperspectral Pansharpening

Tree-Structured Neural Network for Hyperspectral Pansharpening

Fault diagnosis of a computer interlocking system for railway signal control

The signal control of the railway transportation system is crucial for operational safety. This paper briefly introduces the computer interlocking system for railway signal control, describes the tree-structured neural network used for fault diagnosis of the interlocking system, and introduces the particle swarm optimization (PSO) algorithm for improvement. Finally, a simulation experiment was conducted on a railway station to compare the traditional back-propagation neural network (BPNN), the support vector machine, the traditional tree-structured neural network, and the improved tree-structured neural network for fault diagnosis. It was found that the topological structure of the device distribution in the railway station could be transformed into a tree structure, and with the introduction of hidden nodes, it could become a binary tree structure where each leaf node represents a device; the improved tree-structured neural network had the highest recognition performance for both two-class tasks (determining system failure or not) and multi-class tasks (identifying fault type).

Enhancing vulnerability detection via AST decomposition and neural sub-tree encoding

The explosive growth of software vulnerabilities poses a serious threat to the system security and has become one of the urgent problems of the day. However, existing vulnerability detection methods are still faced with limitations in reaching the balance between detection accuracy, efficiency and applicability. Following a divide-and-conquer strategy, this paper proposes TrVD (abstract syntax Tree decomposition based Vulnerability Detector) to disclose the indicative semantics implied in the source code fragments for accurate and efficient vulnerability detection. To facilitate the capture of subtle semantic features, TrVD converts the AST of a code fragment into an ordered set of sub-trees of restricted sizes and depths with a novel decomposition algorithm. The semantics of each sub-tree can thus be effectively collected with a carefully designed tree-structured neural network. Finally, a Transformer-style encoder is utilized to aggregate the long-range contextual semantics of all sub-trees into a vulnerability-specific vector to represent the target code fragment. The extensive experiments conducted on five large datasets consisting of diverse real-world and synthetic vulnerable samples demonstrate the performance superiority of TrVD against SOTA approaches in detecting the presence of vulnerabilities and pinpointing the vulnerability types. The ablation studies also confirm the effectiveness of TrVD’s core designs.

Tree-structured neural networks: Spatiotemporal dynamics and optimal control

How the network topology drives the response dynamic is a basic question that has not yet been fully answered in neural networks. Elucidating the internal relation between topological structures and dynamics is instrumental in our understanding of brain function. Recent studies have revealed that the ring structure and star structure have a great influence on the dynamical behavior of neural networks. In order to further explore the role of topological structures in the response dynamic, we construct a new tree structure that differs from the ring structure and star structure of traditional neural networks. Considering the diffusion effect, we propose a diffusion neural network model with binary tree structure and multiple delays. How to design control strategies to optimize brain function has also been an open question. Thus, we put forward a novel full-dimensional nonlinear state feedback control strategy to optimize relevant neurodynamics. Some conditions about the local stability and Hopf bifurcation are obtained, and it is proved that the Turing instability does not occur. Moreover, for the formation of the spatially homogeneous periodic solution, some diffusion conditions are also fused together. Finally, several numerical examples are carried out to illustrate the results’ correctness. Meanwhile, some comparative experiments are rendered to reveal the effectiveness of the proposed control strategy.

A Recursive tree-structured neural network with goal forgetting and information aggregation for solving math word problems

Most existing state-of-the-art neural network models for math word problems use the Goal-driven Tree-Structured decoder (GTS) to generate expression trees. However, we found that GTS does not provide good predictions for longer expressions, mainly because it does not capture the relationships among the goal vectors of each node in the expression tree and ignores the position order of the nodes before and after the operator. In this paper, we propose a novel Recursive tree-structured neural network with Goal Forgetting and information aggregation (RGFNet) to address these limits. The goal forgetting and information aggregation module is based on ordinary differential equations (ODEs) and we use it to build a sub-goal information feedback neural network (SGIFNet). Unlike GTS, which uses two-layer gated-feedforward networks to generate goal vectors, we introduce a novel sub-goal generation module. The sub-goal generation module could capture the relationship among the related nodes (e.g. parent nodes, sibling nodes) using attention mechanism. Experimental results on two large public datasets i.e. Math23K and Ape-clean show that our tree-structured model outperforms the state-of-the-art models and obtains answer accuracy over 86%. Furthermore, the performance on long-expression problems is promising.11The source code of this paper is available at: https://github.com/SCNU203/RGFNet.

NeuralTree: A 256-Channel 0.227-μJ/Class Versatile Neural Activity Classification and Closed-Loop Neuromodulation SoC.

Closed-loop neural interfaces with on-chip machine learning can detect and suppress disease symptoms in neurological disorders or restore lost functions in paralyzed patients. While high-density neural recording can provide rich neural activity information for accurate disease-state detection, existing systems have low channel counts and poor scalability, which could limit their therapeutic efficacy. This work presents a highly scalable and versatile closed-loop neural interface SoC that can overcome these limitations. A 256-channel time-division multiplexed (TDM) front-end with a two-step fast-settling mixed-signal DC servo loop (DSL) is proposed to record high-spatial-resolution neural activity and perform channel-selective brain-state inference. A tree-structured neural network (NeuralTree) classification processor extracts a rich set of neural biomarkers in a patient- and disease-specific manner. Trained with an energy-aware learning algorithm, the NeuralTree classifier detects the symptoms of underlying disorders (e.g., epilepsy and movement disorders) at an optimal energy-accuracy trade-off. A 16-channel high-voltage (HV) compliant neurostimulator closes the therapeutic loop by delivering charge-balanced biphasic current pulses to the brain. The proposed SoC was fabricated in 65nm CMOS and achieved a 0.227μJ/class energy efficiency in a compact area of 0.014mm2/channel. The SoC was extensively verified on human electroencephalography (EEG) and intracranial EEG (iEEG) epilepsy datasets, obtaining 95.6%/94% sensitivity and 96.8%/96.9% specificity, respectively. In-vivo neural recordings using soft μECoG arrays and multi-domain biomarker extraction were further performed on a rat model of epilepsy. In addition, for the first time in literature, on-chip classification of rest-state tremor in Parkinson's disease (PD) from human local field potentials (LFPs) was demonstrated.

Protein-Protein Interaction Extraction using Attention-based Tree-Structured Neural Network Models

In order to comprehend underlying biological processes, it is necessary to identify interactions between proteins. It is typically quite difficult to extract a protein-protein interaction (PPI) from text data as text data is complex in nature. Unlike sequential models, tree-structured neural network models have the ability to consider syntactic and semantic dependencies between different portions of the text and can provide structural information at the phrase level. This paper investigates tree-structured neural network models for the PPI task and the results show their supremacy over sequential models and their effectiveness for this task.

TreeLSTM with tag-aware hypernetwork for sentence representation

Tree-structured neural networks, such as TreeLSTM and its variants, have proven effective for learning semantic representations of sentences, which are useful for a variety of tasks in natural language processing such as text categorisation, text semantic matching and machine translation. These neural network models take as inputs parse trees of sentences, which are generated by a language parser. However, most existing tree-structured neural network models lack the ability of distinguishing different syntactic compositions, thus the expressive power of these models is limited. Moreover, the syntactic knowledge provided by Part-of-Speech tags in a parse tree has not been fully utilised in existing tree-structured neural network models. It is expected that such syntactic knowledge should help distinguish syntactic compositions, so should result in better semantic representation. This paper proposes a novel neural network model, TagHyperTreeLSTM, which contains two components, a tag-aware hypernetwork and a sentence encoder. The tag-aware hypernetwork, which accepts tags as inputs, generates the parameters of the sentence encoder dynamically in order to distinguish different syntactic compositions. The sentence encoder, which accepts words as inputs, generates the final sentence representation. Experimental results show that the proposed model achieves superior or competitive performance in text classification and text semantic matching based on six benchmark datasets when compared against previous tree-structured models.

Cardinality Estimation of an SQL Query Using Recursive Neural Networks

To learn complex interactions between predicates and accurately estimate the cardinality of an SQL query, we develop a novel framework based on recursive tree-structured neural networks, which take into account the natural properties of logical expressions: compositionality and n-ary associativity. The proposed architecture is an extension of MSCN (multi-set convolutional network) for queries containing both conjunction and disjunction operators. The goal is to represent an arbitrary logical expression in a continuous vector space by combining sub-expression vectors according to the operator type. We compared the proposed approach with the histogram-based approach on the real-world dataset and showed that our approach significantly outperforms histograms with a large margin.

Syntax-aware entity representations for neural relation extraction

Distantly supervised relation extraction has been widely used to find novel relational facts between entities from text, and can be easily scaled to very large corpora. Previous studies on neural relation extraction treat this task as a multi-instance learning problem, and encode the sentences in low-dimensional spaces via neural networks. Although great progress has been made, they seldom consider the information represented by entities, which are of great significance to relation extraction. In this article, we propose several methods based on different tree-based models to learn syntax-aware entity representations for neural relation extraction. First, we encode the context of entities on dependency trees as sentence-level entity embedding based on tree-structured neural network models. Then, we utilize inter-sentence attention mechanism to obtain sentence bag level entity embedding over all sentences containing the specified entity pair. Finally, we combine both sentence embedding and entity embedding for relation classification. Experimental results on a widely used real-world dataset indicate that our system performs better than the state-of-the-art systems of relation extraction.

Tree-Structured Neural Networks With Topic Attention for Social Emotion Classification

Social emotion classification studies the emotion distribution evoked by an article among numerous readers. Although recently neural network-based methods can improve the classification performance compared with the previous word-emotion and topic-emotion approaches, they have not fully utilized some important sentence language features and document topic features. In this paper, we propose a new neural network architecture exploiting both the syntactic information of a sentence and topic distribution of a document. The proposed architecture first constructs a tree-structured long short-term memory (Tree-LSTM) network based on the sentence syntactic dependency tree to obtain a sentence vector representation. For a multi-sentence document, we then use a Chain-LSTM network to obtain the document representation from its sentences’ hidden states. Furthermore, we design a topic-based attention mechanism with two attention levels. The word-level attention is used for weighting words of a single-sentence document and the sentence-level attention for weighting sentences of a multi-sentence document. The experiments on three public datasets show that the proposed scheme outperforms the state-of-the-art ones in terms of higher average Pearson correlation coefficient and MicroF1 performance.

Using Neural Networks to Generate Inferential Roles for Natural Language.

Neural networks have long been used to study linguistic phenomena spanning the domains of phonology, morphology, syntax, and semantics. Of these domains, semantics is somewhat unique in that there is little clarity concerning what a model needs to be able to do in order to provide an account of how the meanings of complex linguistic expressions, such as sentences, are understood. We argue that one thing such models need to be able to do is generate predictions about which further sentences are likely to follow from a given sentence; these define the sentence's “inferential role.” We then show that it is possible to train a tree-structured neural network model to generate very simple examples of such inferential roles using the recently released Stanford Natural Language Inference (SNLI) dataset. On an empirical front, we evaluate the performance of this model by reporting entailment prediction accuracies on a set of test sentences not present in the training data. We also report the results of a simple study that compares human plausibility ratings for both human-generated and model-generated entailments for a random selection of sentences in this test set. On a more theoretical front, we argue in favor of a revision to some common assumptions about semantics: understanding a linguistic expression is not only a matter of mapping it onto a representation that somehow constitutes its meaning; rather, understanding a linguistic expression is mainly a matter of being able to draw certain inferences. Inference should accordingly be at the core of any model of semantic cognition.

A discourse-aware neural network-based text model for document-level text classification

Capturing semantics scattered across entire text is one of the important issues for Natural Language Processing (NLP) tasks. It would be particularly critical with long text embodying a flow of themes. This article proposes a new text modelling method that can handle thematic flows of text with Deep Neural Networks (DNNs) in such a way that discourse information and distributed representations of text are incorporate. Unlike previous DNN-based document models, the proposed model enables discourse-aware analysis of text and composition of sentence-level distributed representations guided by the discourse structure. More specifically, our method identifies Elementary Discourse Units (EDUs) and their discourse relations in a given document by applying Rhetorical Structure Theory (RST)-based discourse analysis. The result is fed into a tree-structured neural network that reflects the discourse information including the structure of the document and the discourse roles and relation types. We evaluate the document model for two document-level text classification tasks, sentiment analysis and sarcasm detection, with comparisons against the reference systems that also utilise discourse information. In addition, we conduct additional experiments to evaluate the impact of neural network types and adopted discourse factors on modelling documents vis-à-vis the two classification tasks. Furthermore, we investigate the effects of various learning methods, input units on the quality of the proposed discourse-aware document model.

Type-Aware Question Answering over Knowledge Base with Attention-Based Tree-Structured Neural Networks

Question answering (QA) over knowledge base (KB) aims to provide a structured answer from a knowledge base to a natural language question. In this task, a key step is how to represent and understand the natural language query. In this paper, we propose to use tree-structured neural networks constructed based on the constituency tree to model natural language queries. We identify an interesting observation in the constituency tree: different constituents have their own semantic characteristics and might be suitable to solve different subtasks in a QA system. Based on this point, we incorporate the type information as an auxiliary supervision signal to improve the QA performance. We call our approach type-aware QA. We jointly characterize both the answer and its answer type in a unified neural network model with the attention mechanism. Instead of simply using the root representation, we represent the query by combining the representations of different constituents using task-specific attention weights. Extensive experiments on public datasets have demonstrated the effectiveness of our proposed model. More specially, the learned attention weights are quite useful in understanding the query. The produced representations for intermediate nodes can be used for analyzing the effectiveness of components in a QA system.

A Comparison of Neural Classifiers for Graffiti Recognition

Technological advances and the enormous flood of papers have motivated many researchers and companies to innovate new technologies. In particular, handwriting recognition is a very useful technology to support applications like electronic books (eBooks), post code readers (that sort mails in post offices), and some bank applications. This paper proposes three systems to discriminate handwritten graffiti digits (0 to 9) and some commands with different architectures and abilities. It introduces three classifiers, namely single neural network (SNN) classifier, parallel neural networks (PNN) classifier and tree-structured neural network (TSNN) classifier. The three classifiers have been designed through adopting feed forward neural networks. In order to optimize the network parameters (connection weights), the back-propagation algorithm has been used. Several architectures are applied and examined to present a comparative study about these three systems from different perspectives. The research focuses on examining their accuracy, flexibility and scalability. The paper presents an analytical study about the impacts of three factors on the accuracy of the systems and behavior of the neural networks in terms of the number of the hidden neurons, the model of the activation functions and the learning rate. Therefore, future directions have been considered significantly in this paper through designing particularly flexible systems that allow adding many more classes in the future without retraining the current neural networks.

A neural tree and its application to spam e-mail detection

This paper presents a new approach to constructing a neural tree to integrate the advantages of decision trees and neural networks. The proposed neural tree, called a quadratic-neuron-based neural tree (QUANT), is a tree-structured neural network composed of neurons with quadratic neural-type junctions for pattern classification. A quadratic neuron is capable of forming a hyper-ellipsoid that can be varied in sizes and in locations on the space spanned by the input variables. Via a batch-mode training algorithm, the QUANT grows a neural tree containing quadratic neurons in its nodes. These quadratic neurons recursively partition the feature space into hyper-ellipsoidal-shaped sub-regions. The QUANT has the partial incremental capability so that it does not need to re-construct a new neural tree to accommodate new training data whenever new data are introduced to a trained QUANT. To demonstrate the performance of the proposed QUANT, one pattern recognition problem and the spam e-mail detection problem were tested.

트리구조 신경망을 이용한 냉연 강판 표면 결함의 분류

A new tree-structured neural network classifier is proposed for the automatic real-time inspection of cold-rolled steel strip surface defects. The defects are classified into 3 groups such as area type, disk type, area & line type in the first stage of the tree-structured neural network. The defects are classified in more detail into 11 major defect types which are considered as serious defects in the second stage of neural network. The tree-structured neural network classifier consists of 4 different neural networks and optimum features are selected for each neural network classifier by using SFFS algorithm and correlation test. The developed classifier demonstrates very plausible result which is compatible with commercial products having high world-wide market shares.

Myoelectric control of a computer animated hand: A new concept based on the combined use of a tree-structured artificial neural network and a data glove

This paper proposes a new learning set-up in the field of control systems for multifunctional hand prostheses. Two male subjects with a traumatic one-hand amputation performed simultaneous symmetric movements with the healthy and the phantom hand. A data glove on the healthy hand was used as a reference to train the system to perform natural movements. Instead of a physical prosthesis with limited degrees of freedom, a virtual (computer-animated) hand was used as the target tool. Both subjects successfully performed seven different motoric actions with the fingers and wrist. To reduce the training time for the system, a tree-structured, self-organizing, artificial neural network was designed. The training time never exceeded 30 seconds for any of the configurations used, which is three to four times faster than most currently used artificial neural network (ANN) architectures.

HIERARCHICAL CLUSTERING FOR COMPLEX DATA

In this paper we introduce a new tree-structured self-organizing neural network called a dynamical growing self-organizing tree (DGSOT). This DGSOT algorithm constructs a hierarchy from top to bottom by division. At each hierarchical level, the DGSOT optimizes the number of clusters, from which the proper hierarchical structure of the underlying data set can be found. We propose a K-level up distribution (KLD) mechanism. This KLD scheme increases the scope for data distribution in the hierarchy, which allows the data mis-clustered in the early stages to be re-evaluated at a later stage increasing the accuracy of the final clustering result. The DGSOT algorithm, combined with the KLD mechanism, overcomes the drawbacks of traditional hierarchical clustering algorithms (e.g., hierarchical agglomerative clustering). The DGSOT algorithm has been tested on two benchmark data sets including gene expression complex data set and we observe that our algorithm extracts patterns with different levels of abstraction. Furthermore, our approach is useful on recognizing features in complex gene expression data. As a dendrogram, these results can be easily displayed for visualization.

Algorithm for building a neural network for function approximation

A technique for constructing a multilayer tree-structured neural network, which provides a continuous, piece-wise linear function approximation, is presented. The method uses a growth network with linear threshold neurons. Neurons are added to a binary tree until the approximation error at all sampling points is brought down to within a specified ±Δ. The number of neurons in the constructed network depends on the samples provided as well as the specified tolerance Δ, thus enabling a trade-off between accuracy and network size. In comparison to approaches such as backpropagation, the proposed technique requires no assumptions regarding the number of neurons, learning rate, momentum term, or initial weight values. It also does not suffer from problems of local minima. Examples are presented to illustrate the effectiveness of the technique.